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ABSTRACT 


The  progress  over  a one  year  period  on  a program  to  evaluate  vari- 
ous Navy  operation  oriented  decision  aids  is  summarized.  The  work  over 
the  period  included;  (1)  evaluation  perspective  development,  (2)  laboratory 
coordination,  (3)  evaluation  method  development  for  a strike  timing  aid  and 
implementation  of  the  evaluation  design,  (4)  report  preparation,  and  (5)  ini- 
tial steps  toward  the  evaluation  of  a second  operational  decision  aid.  The 
work  in  each  of  these  areas  is  summarized. 
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DECISION  AID  EVALUATION 


Over  the  past  year,  the  Applied  Psychological  Services  has  been  en- 
gaged in  a program  to  evaluate  decision  aids,  developed  by  others,  under  the 
Office  of  Naval  Research's  operational  decision  aid  development  program. 
Specifically,  for  each  member  of  a set  of  individual  operational  decision  aids, 
the  purpose  of  the  present  program  is  to  answer  such  questions  as; 

• Does  it  work? 

• Why  does  it  work? 

• How  can  it  be  made  to  work  better? 


The  evaluations  are  to  be  conducted  in  accordance  with  the  following  criteria: 

• use  of  well -controlled,  precise,  multivariate 

methods 

• programmatic  approach 

• full  coordination  with  ONR 

• orientation  towards  possible  conditions  of 

actual  aid  use  in  the  Navy 

• coordination  with  aid  developers  but  main- 

tenance of  evaluation  integrity 

• use  of  previous  developed  action  scenarios 

where  possible 


This  Annual  Report- -Decision  Aid  Evaluation  Program  describes  the 
work  performed  during  the  period  July  1978  through  June  1979.  The  report 
is  summary  in  nature  because  a full  technical  report  describing  the  details  of 
the  major  accomplishment  over  the  period,  an  evaluation  of  the  strike  timing 
decision  aid  developed  by  Analytics,  Inc. , is  currently  in  preparation. 

The  activities  over  the  year  may  be  subdivided  over  five  areas: 

(1)  evaluation  perspective  development,  (2)  coordination  with  personnel  re- 
sponsible for  the  development  of  the  operational  decision  aid  laboratory  at 
the  Wharton  School,  University  of  Pennsylvania,  (3)  development  of  a plan  for 
evaluating  the  strike  timing  aid  developed  by  Analytics,  Inc.  and  collection  of 
the  required  evaluative  data,  (4)  report  preparation  relative  to  the  strike  tim- 
ing aid  evaluation,  and  (5)  preparation  for  an  evaluation  of  the  emission  control 
decision  aid  developed  by  Decision  Science  Applications,  Inc.  The  work  com- 
pleted in  each  of  these  areas  is  described  categorically  below. 


Evaluation  Perspective  Development 

Because  it  seemed  important  to  place  the  present  evaluation  program 
in  the  context  of  a total  decision  aid  developmental  framework,  a conception 
of  the  steps  to  be  followed  during  the  development  of  such  aids  was  developed. 
Figure  1 presents  the  conception.  The  figure  is  read  from  bottom  to  top  with 
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Figure  1.  Sequence  of  aid  development. 
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the  considerations  involved  in  each  stage  entering  from  the  left  of  each  box 
and  the  results  of  each  stage  exiting  to  the  right.  The  arrows  emerging 
from  the  top  of  each  Figure  1 box  represent  criteria  which  may  be  applied 
after  each  developmental  stage.  These  criteria  are  defined  in  Table  1.  The 
rounded  boxes  associated  with  each  rectangular,  stage  box  represent  descrip- 
tors which  may  be  applied  as  the  criteria  at  the  successive  stages  are  met. 
Accordingly,  an  aid  maybe  successively  called  "suitable','  "testable,"  "rea- 
sonable," "valid,"  "effective,"  and  "useful."  Note  that  we  are  primarily  con- 
cerned within  the  present  program  with  the  upper  right  box- -"validation  testing  - 
exercise  in  lab  experiment  and  compare  with  intermediate  criterion." 


Coordination  with  Laboratory  Site 


The  data  acquisition  aspects  of  the  aid  evaluations  are  completed  at 
the  decision  aid  laboratory,  established  by  the  Office  of  Naval  Research  (ONR), 
at  the  Wharton  School,  University  of  Pennsylvania.  Ongoing  coordination  was 
completed  with  the  personnel  at  the  laboratory.  Such  coordination  served  to 
assure  that  the  laboratory's  capability  was  sufficient  to  support  anticipated 
aid  evaluations.  To  this  end,  considerable  progress  was  made,  in  collabora- 
tion with  Wharton  School  personnel,  towards  increasing  the  reliability  of  the 
equipment  suite,  decreasing  access  time  in  the  time  sharing  situation,  and 
organizing  an  effective  data  recording  and  acquisition  system.  While  some 
problems  still  remain  in  each  of  these  areas,  considerable  progress  was  made 
towards  minimizing  their  effects  on  any  emergent  evaluative  data. 


Strike  Timing  Aid  Evaluation 


The  strike  timing  aid,  developed  by  Analytics,  Inc. , was  the  first  aid 
to  be  evaluated  under  the  present  program.  The  strike  timing  aid  is  a sophis- 
ticated, man-computer  interactive  tool  for  deriving  strike  timing  decisions. 

It  considers  a host  of  own  force  readiness,  enemy  force,  and  environmental 
considerations  in  a probabilistic  manner  and  yields  the  information  necessary 
for  deriving  a "preferred"  strike  launch  time. 

To  evaluate  the  strike  timing  aid,  an  evaluative  plan  was  developed  and 
distributed  to  all  concerned  parties  (Analytics,  Inc. , Wharton  School,  Office 
of  Naval  Research).  Because  there  was  a wide  variety  of  comments — some- 
times conflicting — to  the  evaluative  plan,  a coordination  meeting  was  called  at 
which  the  various  concerned  organizations  were  represented.  As  the  result  of 
the  coordination  meeting,  a second  plan  was  written  to  reflect  the  various  sug- 
gested modifications.  This  second  plan  was  submitted  to  the  concerned  organ- 
izations with  the  result  that  a new  set  of  comments  (sometimes  conflicting  with 
original  remarks)  was  received.  A second  coordination  meeting  resolved  the 
issues  and  a final  plan  was  written  and  submitted. 
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Criteria  for  Evaluating  Decision  Aids  at  Various  Developmental  Stages 
Criterion  Definitions 


Internal  consistency 


Indifference  to  trivial  aggregation 


Correct  prediction  in  the  extreme 
(predictive  or  empirical  validity) 

Correct  prediction  in  mid  range 
(predictive  or  empirical  validity) 

Construct  validity 

Content  (variable/  parameter) 
validity  ( f idelity) 

Realism  or  "face  validity" 
Richness  of  output 
Ease  of  use 


Extent  to  which  the  constructs  of  the  aid 
are  marked  by  coherence  and  similarity 
of  treatment 

Potential  of  the  aid  to  avoid  major  changes 
in  output  when  input  groupings  or  conditions 
undergo  insignificant  fluctuations 

Extent  of  agreement  (correctness  of  predic- 
tions) between  aid  and  actual  performance 
at  very  high/ low  values  of  conditions 

Like  above  for  middle  ranges  values  of 
conditions 

Theoretic  adequacy  of  the  aid  constructs 

Extent  to  which  the  aid's  variables/ para- 
meters match  real  life  conditions 

Extent  to  which  selected  content  matches 
each  attribute  modeled 

Number  and  type  of  output  variables  and 
forms  of  presentation 

Extent  to  which  an  analyst  can  readily  pre- 
pare data  for,  apply  and  extract  understand- 
able results  from  the  aid 


Cost  of  development 
Transportability  - generality 
Cost  of  use 


Internal  validity 


Value  of  effort  to  conceive,  develop,  test, 
document,  and  support 

Extent  to  which  different  systems,  missions, 
and  configurations  can  be  considered 

Value  of  all  effort  involving  use  of  aid  in- 
cluding data  collection,  input,  data  proc- 
essing, and  analysis  of  results 

Extent  to  which  outputs  are  repeatable  with- 
out change  of  inputs 


Event  or  time  series  validity 


Hypothesis  validity 


Extent  to  which  simulation  predicts  events, 
event  patterns 

Extent  to  which  model  relationships  corre- 
spond to  similar  relationships  in  the  observ 
able  universe 
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Then,  work  started  in  accordance  with  the  agreed  on  plan.  Because 
the  plan  called  for  comparing  the  "correctness"  of  strike  timing  decisions 
made  under  various  levels  of  aid  employment,  the  most  preferred  answer  to 
a set  of  stimulus  strike  timing  problems  was  required.  To  this  end,  a panel 
of  Navy  officers  was  organized  at  the  Office  of  Naval  Research.  The  panel 
was  asked  to  rank  order  the  members  of  a set  of  possible  strike  launch  times 
for  each  problem.  A modified  Delphi  technique  was  employed  to  collect  the 
information  and  group  convergence  was  not  difficult  to  achieve.  The  data 
collection  took  place  over  a 1.5  day  period.  One  interesting  artifact  of  the 
work  was  the  finding  that  the  panel  members  themselves  often  possessed  im- 
plicit and  explicit  personal  algorithms  for  solving  such  problems. 

The  general  evaluative  design  for  the  strike  timing  aid  is  shown  in 
Figure  2.  There  are  five  levels  of  aid  employment  (aid  variations),  two  prob- 
lem difficulty  levels,  and  two  levels  of  operational  experience. 

To  train  the  evaluation  subjects  in  strike  timing  aid  use,  a separate 
script  was  written  for  each  condition  and  a television  tape  was  prepared  on 
the  basis  of  each  script.  The  50  subjects,  required  by  the  design,  were  re- 
cruited locally.  The  subjects  of  the  little  or  no  operational  experience  group 
were  recruited  from  the  NROTC  group  at  the  University  of  Pennsylvania  while 
the  subjects  of  the  operationally  experienced  group  were  persons  with  recent 
Navy  experience.  They  held  ranks  from  Lieutenant  to  Captain  on  discharge 
from  active  duty.  All  of  the  latter  group  possessed  flight  experience. 

Data  collection  for  each  subject  was  completed  in  a single,  four  hour 
session.  At  the  conclusion  of  the  formal  data  acquisition,  an  extensive  inter- 
view was  completed  with  each  experimental  subject.  The  interview  inquired 
into  perception  of  the  utility  of  the  aid,  areas  for  aid  improvement,  features 
in  the  aid  which  might  be  modified,  subjective  confidence  in  answers  and  the 
like. 


A number  of  formal  analyses  of  the  data  was  also  completed.  The 
results  of  a set  of  variance  analyses  indicated  support  for  the  aid.  A multi- 
attribute utility  analysis  also  yielded  results  which  support  the  aid. 


Report  Preparation 

A full  technical  report  describing  the  methods,  procedures,  and  re- 
sults of  the  evaluation  of  the  strike  timing  aid  was  partially  prepared.  How- 
ever, completion  of  the  report  was  postponed  so  that  the  research  team  could 
focus  its  attention  on  the  second  aid  to  be  evaluated,  the  emission  control  aid 
by  Decision  Science  Applications,  Inc. 


Evaluation  of  the  Emission  Control  Aid 

The  purpose  of  the  emission  control  aid  is  to  help  in  the  derivation  of 
an  electronic  emission  complement  which  will  maximize  the  surveillance 
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Figure  2.  Evaluative  paradigm  for  strike  timing  aid 


effectiveness  of  a task  force  while  minimizing  the  amount  of  information 
about  the  task  force  which  is  made  available  to  the  enemy.  Like  the  strike 
timing  aid,  the  emission  control  aid  is  man-computer  interactive  and  is  based 
on  extensive  sets  of  optimization  algorithms. 


Each  aid  evaluation  seems  to  possess  special  problems  which  do  not 
become  evidenced  on  initial  exposures  to  the  aid  or  initial  sets  of  interactions 
with  the  aid's  developers.  In  the  case  of  the  emission  control  aid,  there  were 
two  such  problems;  (1)  the  full  utility  of  the  aid  may  not  be  assessable  unless 
the  user's  are  provided  with  hard  copy  of  their  various  emission  control  conjee-  ^ 
tures  about  each  problem,  and  (2)  the  full  utility  of  the  aid  cannot  be  assessed 
unless  the  users  have  considerable  training  related  to  a set  of  heuristic  rules 
(believed  by  the  aid  developer  to  be  important  to  emission  control)  for  proper 
emission  control.  i 


The  first  of  these  problems  was  managed  by  adding  a Tektronix  4013 
with  a hard  copy  unit  to  the  research  equipment.  The  second  problem,  heuris 
tic  training,  was  managed  by  building  an  extensive  training  period  into  the 
aid  evaluation. 
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The  general  design  for  the  emission  control  aid  evaluation  is  shown  in 
Figure  3.  This  design  was  derived  as  the  result  of  a preliminary  design  devel- 
oped by  Applied  Psychological  Services  and  a set  of  coordination  meetings  in- 
volving Applied  Psychological  Services,  Decision  Science  Applications,  Wharton 
School,  and  Office  of  Naval  Research  personnel.  The  full  design  was  contained 
in  a formal  report  submitted  on  8 June  1979. 

By  the  conclusion  of  the  period  reviewed  in  this  Annual  Report,  the 
laboratory  preparation  aspects  (problem  insertion  and  test,  problem  correction, 
programming,  and  equipment  organization)  were  about  75  per  cent  completed. 

As  is  not  unusual,  considerable  computer  program  reorganization/ correction 
was  necessary.  The  training  program  was  detailed  and  arrangements  for  sub- 
jects were  partially  completed.  According  to  current  plans,  as  many  subjects 
as  possible  will  be  drawn  from  instructor  and  staff  personnel,  with  operations 
experience,  at  the  U.  S.  Naval  Academy.  The  training  will  be  administered  at 
the  Academy  while  the  formal  evaluative  data  acquisition  will  take  place  at  the 
decision  aid  laboratory  at  the  Wharton  School.  Additional  subjects  will  be  re- 
cruited and  trained  locally,  if  necessary.  According  to  current  plans,  the 
training  for  the  Academy  will  take  place  during  the  weeek  of  9 July  and  data  ac- 
quisition will  start  immediately  after  the  1.5  day  training  period  is  completed. 
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Figure  3.  Design  for  the  EWAR  decision  aid  evaluatioiu 


Reports 


Over  the  period,  two  design  reports  were  prepared; 

Applied  Psychological  Services.  Evaluative  design  for 
EWAR  decision  aid.  Wayne,  Pa. ; Applied  Psycho- 
logical Services,  1979. 

Siegel,  A.  I.  An  experimental  design  for  evaluating  the 
’’strike  timing  aid."  Wayne,  Pa^;  Applied  Psycho- 
logical  Services,  1978. 


Personnel  assigned,  over  the  period,  to  the  work  were;  Dr.  A.  I. 
Siegel  (Principal  Investigator)  and  Dr.  E.  Madden  (Research  Associate). 
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